Least Squares and Entropy as Penalty Functions

Mathematical measures of entropy as defined by Shannon (1948) and Kullback and Leibler (1951) are currently in vogue in the field of econometrics, primarily due to the comprehensive work by Golan, Judge, and Miller (1996). In this paper, an alternative interpretation of the entropy measure as a penalty function over deviations is presented. Using this interpretation, a number of parallels are drawn with least squares estimators, and it is demonstrated that, with a minor modification of the traditional least squares estimator, both approaches may be applied to the general linear model. The advantages and disadvantages of each approach are discussed, and a philosophical approach to the selection of estimation technique is suggested.